Assessing the State of Self-Supervised Human Activity Recognition using Wearables

The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems for scenarios where only small amounts of labeled training samples can be collected. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone hand-crafted features for the classic activity recognition chain. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, such that each dimension captures specific aspects of performance, including the robustness to differing source and target conditions, the influence of dataset characteristics, and the feature space characteristics. We utilize this framework to assess seven state-of-the-art self-supervised methods for HAR, leading to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios.Comment: update

On the Benefit of Generative Foundation Models for Human Activity Recognition

In human activity recognition (HAR), the limited availability of annotated data presents a significant challenge. Drawing inspiration from the latest advancements in generative AI, including Large Language Models (LLMs) and motion synthesis models, we believe that generative AI can address this data scarcity by autonomously generating virtual IMU data from text descriptions. Beyond this, we spotlight several promising research pathways that could benefit from generative AI for the community, including the generating benchmark datasets, the development of foundational models specific to HAR, the exploration of hierarchical structures within HAR, breaking down complex activities, and applications in health sensing and activity summarization.Comment: Generative AI for Pervasive Computing (GenAI4PC) Symposium within UbiComp/ISWC 202

Fine-grained Human Activity Recognition Using Virtual On-body Acceleration Data

Previous work has demonstrated that virtual accelerometry data, extracted from videos using cross-modality transfer approaches like IMUTube, is beneficial for training complex and effective human activity recognition (HAR) models. Systems like IMUTube were originally designed to cover activities that are based on substantial body (part) movements. Yet, life is complex, and a range of activities of daily living is based on only rather subtle movements, which bears the question to what extent systems like IMUTube are of value also for fine-grained HAR, i.e., When does IMUTube break? In this work we first introduce a measure to quantitatively assess the subtlety of human movements that are underlying activities of interest--the motion subtlety index (MSI)--which captures local pixel movements and pose changes in the vicinity of target virtual sensor locations, and correlate it to the eventual activity recognition accuracy. We then perform a "stress-test" on IMUTube and explore for which activities with underlying subtle movements a cross-modality transfer approach works, and for which not. As such, the work presented in this paper allows us to map out the landscape for IMUTube applications in practical scenarios

Towards Using Unlabeled Data in a Sparse-coding Framework for Human Activity Recognition

We propose a sparse-coding framework for activity recognition in ubiquitous and mobile computing that alleviates two fundamental problems of current supervised learning approaches. (i) It automatically derives a compact, sparse and meaningful feature representation of sensor data that does not rely on prior expert knowledge and generalizes extremely well across domain boundaries. (ii) It exploits unlabeled sample data for bootstrapping effective activity recognizers, i.e., substantially reduces the amount of ground truth annotation required for model estimation. Such unlabeled data is trivial to obtain, e.g., through contemporary smartphones carried by users as they go about their everyday activities. Based on the self-taught learning paradigm we automatically derive an over-complete set of basis vectors from unlabeled data that captures inherent patterns present within activity data. Through projecting raw sensor data onto the feature space defined by such over-complete sets of basis vectors effective feature extraction is pursued. Given these learned feature representations, classification backends are then trained using small amounts of labeled training data. We study the new approach in detail using two datasets which differ in terms of the recognition tasks and sensor modalities. Primarily we focus on transportation mode analysis task, a popular task in mobile-phone based sensing. The sparse-coding framework significantly outperforms the state-of-the-art in supervised learning approaches. Furthermore, we demonstrate the great practical potential of the new approach by successfully evaluating its generalization capabilities across both domain and sensor modalities by considering the popular Opportunity dataset. Our feature learning approach outperforms state-of-the-art approaches to analyzing activities in daily living.Comment: 18 pages, 12 figures, Pervasive and Mobile Computing, 201

The ambient kitchen: a pervasive sensing environment for situated services

In this paper we describe the demonstration of the Ambient Kitchen, a pervasive sensing environment designed for improving cooking skills, promoting healthier eating, and helping cognitively impaired people to live more independent in their own homes. The kitchen is instrumented with an embedded sensing infrastructure including RFID, Newcastle University Culture lab’s proprietary wireless accelerometers (WAX), microphone, camera, pressure sensors and tablet computers. Several applications including real-time activity recognition, recipe displays, and real-time food recognition are deployed in our kitchen

Cardiorespiratory fitness is associated with hard and light intensity physical activity but not time spent sedentary in 10–14 year old schoolchildren: the HAPPY study

Sedentary behaviour is a major risk factor for developing chronic diseases and is associated with low cardiorespiratory fitness in adults. It remains unclear how sedentary behaviour and different physical activity subcomponents are related to cardiorespiratory fitness in children. The purpose of this study was to assess how sedentary behaviour and different physical activity subcomponents are associated with 10–14 year-old schoolchildren's cardiorespiratory fitness

Video Based Assessment of OSATS Using Sequential Motion Textures

Presented at the Fifth Workshop on Modeling and Monitoring of Computer Assisted Interventions (M2CAI)We present a fully automated framework for video based surgical skill assessment that incorporates the sequential and qualitative aspects of surgical motion in a data-driven manner. We replicate Objective Structured Assessment of Technical Skills (OSATS) assessments, which provides both an overall and in-detail evaluation of basic suturing skills required for surgeons. Video analysis techniques are introduced that incorporate sequential motion aspects into motion textures. We also demonstrate significant performance improvements over standard bag-of-words and motion analysis approaches. We evaluate our framework in a case study that involved medical students with varying levels of expertise performing basic surgical tasks in a surgical training lab setting.Intuitive Surgica

Occupancy monitoring using environmental & context sensors and a hierarchical analysis framework

Saving energy in residential and commercial buildings is of great interest due to diminishing resources. Heating ventilation and air conditioning systems, and electric lighting are responsible for a significant share of energy usage, which makes it desirable to optimise their operations while maintaining user comfort. Such optimisation requires accurate occupancy estimations. In contrast to current, often invasive or unreliable methods we present an approach for accurate occupancy estimation using a wireless sensor network (WSN) that only collects non-sensitive data and a novel, hierarchical analysis method. We integrate potentially uncertain contextual information to produce occupancy estimates at different levels of granularity and provide confidence measures for effective building management. We evaluate our framework in real-world deployments and demonstrate its effectiveness and accuracy for occupancy monitoring in both low-and high-traffic area scenarios. Furthermore, we show how the system is used for analysing historical data and identify effective room misuse and thus a potential for energy saving